Qualitative Aspects in Numerical Data: Structure, Interaction, and Correlation

Entropy and information are common measures of probabilistic models of data, frequently used for discrete and discretized but more rarely for continuous data. We employ finite mixture models, which handle continuous and discrete data simultaneously, to construct a probabilistic model whose entropy can be estimated. Analytic estimates of entropy are intractable for some models, so an approximate sample estimate of entropy is used. A simplified formulation of bootstrap is employed to assess the distribution of entropy, which can then be represented with confidence intervals. We show how entropy can be used as a unified approach to quantifying three fundamental qualitative aspects of probabilistic models: the correlation aspect that corresponds to the linearities in the model, the structure aspect that helps capture model’s nonlinearities, and the interaction aspect underlying the entanglements of attributes resulting from structure or correlation or both.